Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods

Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods

Exploring the sensitivity of Northern Hemisphere atmospheric circulation to different surface temperature forcing using a statistical-dynamical atmospheric model

Climate and weather conditions in the mid-latitudes are strongly driven by the large-scale atmosphere circulation. Observational data indicate that important components of the large-scale circulation have changed in recent decades, including the strength and the width of the Hadley cell, jets, storm tracks and planetary waves. Here, we use a new statistical-dynamical atmosphere model (SDAM) to test the individual sensitivities of the large-scale atmospheric circulation to changes in the zonal temperature gradient, meridional temperature gradient and global-mean temperature. We analyze the Northern Hemisphere Hadley circulation, jet streams, storm tracks and planetary waves by systematically altering the zonal temperature asymmetry, the meridional temperature gradient and the global-mean temperature. Our results show that the strength of the Hadley cell, storm tracks and jet streams depend, in terms of relative changes, almost linearly on both the global-mean temperature and the meridional temperature gradient, whereas the zonal temperature asymmetry has little or no influence. The magnitude of planetary waves is affected by all three temperature components, as expected from theoretical dynamical considerations. The width of the Hadley cell behaves nonlinearly with respect to all three temperature components in the SDAM. Moreover, some of these observed large-scale atmospheric changes are expected from dynamical equations and are therefore an important part of model validation

Discovering dynamics and parameters of nonlinear oscillatory and chaotic systems from partial observations

Despite rapid progress in live-imaging techniques, many complex biophysical and biochemical systems remain only partially observable, thus posing the challenge to identify valid theoretical models and estimate their parameters from an incomplete set of experimentally accessible time series. Here, we combine sensitivity methods and VoteFair popularity ranking to construct an automated hidden dynamics inference framework that can discover predictive nonlinear dynamical models for both observable and latent variables from noise-corrupted incomplete data in oscillatory and chaotic systems. After validating the framework for prototypical FitzHugh-Nagumo oscillations, we demonstrate its applicability to experimental data from squid neuron activity measurements and Belousov-Zhabotinsky (BZ) reactions, as well as to the Lorenz system in the chaotic regime.Comment: 37 pages, 18 figure

Exploring the sensitivity of Northern Hemisphere atmospheric circulation to different surface temperature forcing using a statistical–dynamical atmospheric model

Climate and weather conditions in the mid-latitudes are strongly driven by the large-scale atmosphere circulation. Observational data indicate that important components of the large-scale circulation have changed in recent decades, including the strength and the width of the Hadley cell, jets, storm tracks and planetary waves. Here, we use a new statistical–dynamical atmosphere model (SDAM) to test the individual sensitivities of the large-scale atmospheric circulation to changes in the zonal temperature gradient, meridional temperature gradient and global-mean temperature. We analyze the Northern Hemisphere Hadley circulation, jet streams, storm tracks and planetary waves by systematically altering the zonal temperature asymmetry, the meridional temperature gradient and the global-mean temperature. Our results show that the strength of the Hadley cell, storm tracks and jet streams depend, in terms of relative changes, almost linearly on both the global-mean temperature and the meridional temperature gradient, whereas the zonal temperature asymmetry has little or no influence. The magnitude of planetary waves is affected by all three temperature components, as expected from theoretical dynamical considerations. The width of the Hadley cell behaves nonlinearly with respect to all three temperature components in the SDAM. Moreover, some of these observed large-scale atmospheric changes are expected from dynamical equations and are therefore an important part of model validation.</p

Dynamics, scaling behavior, and control of nuclear wrinkling

The cell nucleus is enveloped by a complex membrane, whose wrinkling has been implicated in disease and cellular aging. The biophysical dynamics and spectral evolution of nuclear wrinkling during multicellular development remain poorly understood due to a lack of direct quantitative measurements. Here, we combine live-imaging experiments, theory, and simulations to characterize the onset and dynamics of nuclear wrinkling during egg development in the fruit fly, Drosophila melanogaster, when nurse cell nuclei increase in size and display stereotypical wrinkling behavior. A spectral analysis of three-dimensional high-resolution data from several hundred nuclei reveals a robust asymptotic power-law scaling of angular fluctuations consistent with renormalization and scaling predictions from a nonlinear elastic shell model. We further demonstrate that nuclear wrinkling can be reversed through osmotic shock and suppressed by microtubule disruption, providing tunable physical and biological control parameters for probing mechanical properties of the nuclear envelope. Our findings advance the biophysical understanding of nuclear membrane fluctuations during early multicellular development.Comment: Main text: 10 pages, 3 figures. SI: 19 pages, 10 figures, 1 tabl

Locally rigid, vessel-based registration for laparoscopic liver surgery

Purpose: Laparoscopic liver resection has significant advantages over open surgery due to less patient trauma and faster recovery times, yet is difficult for most lesions due to the restricted field of view and lack of haptic feedback. Image guidance provides a potential solution but is challenging in a soft deforming organ such as the liver. In this paper, we therefore propose a laparoscopic ultrasound (LUS) image guidance system and study the feasibility of a locally rigid registration for laparoscopic liver surgery. Methods: We developed a real-time segmentation method to extract vessel centre points from calibrated, freehand, electromagnetically tracked, 2D LUS images. Using landmark-based initial registration and an optional iterative closest point (ICP) point-to-line registration, a vessel centre-line model extracted from preoperative computed tomography (CT) is registered to the ultrasound data during surgery. Results: Using the locally rigid ICP method, the RMS residual error when registering to a phantom was 0.7 mm, and the mean target registration error (TRE) for two in vivo porcine studies was 3.58 and 2.99 mm, respectively. Using the locally rigid landmark-based registration method gave a mean TRE of 4.23 mm using vessel centre lines derived from CT scans taken with pneumoperitoneum and 6.57 mm without pneumoperitoneum. Conclusion: In this paper we propose a practical image-guided surgery system based on locally rigid registration of a CT-derived model to vascular structures located with LUS. In a physical phantom and during porcine laparoscopic liver resection, we demonstrate accuracy of target location commensurate with surgical requirements. We conclude that locally rigid registration could be sufficient for practically useful image guidance in the near future

Analytical, Optimal, and Sparse Optimal Control of Traveling Wave Solutions to Reaction-Diffusion Systems

This work deals with the position control of selected patterns in reaction-diffusion systems. Exemplarily, the Schl\"{o}gl and FitzHugh-Nagumo model are discussed using three different approaches. First, an analytical solution is proposed. Second, the standard optimal control procedure is applied. The third approach extends standard optimal control to so-called sparse optimal control that results in very localized control signals and allows the analysis of second order optimality conditions.Comment: 22 pages, 3 figures, 2 table

Database-Based Estimation of Liver Deformation under Pneumoperitoneum for Surgical Image-Guidance and Simulation

The insufflation of the abdomen in laparoscopic liver surgery leads to significant deformation of the liver. The estimation of the shape and position of the liver after insufflation has many important applications, such as providing surface-based registration algorithms used in image guidance with an initial guess and realistic patient-specific surgical simulation. Our proposed algorithm computes a deformation estimate for a patient subject from a database of known insufflation deformations, as a weighted average. The database is built from pre-operative and intra-operative 3D image segmentations. The estimation pipeline also comprises a biomechanical simulation to incorporate patient-specific boundary conditions (BCs) and eliminate any non-physical deformation arising from the computation of the deformation as a weighted average. We have evaluated the accuracy of our intra-subject registration, used for the computation of the displacements stored in the database, and our liver deformation predictions based on segmented, in-vivo porcine CT image data from 5 animals and manually selected vascular landmarks. We found root mean squared (RMS) target registration errors (TREs) of 2.96-11.31mm after intra-subject registration. For our estimated deformation, we found an RMS TRE of 5.82-11.47mm for four of the subjects, on one outlier subject the method failed